Cluster databases provide location transparency to data by allowing multiple systems to serve the same database. One specific type of cluster database is the Oracle Real Application Clusters product, licensed by Oracle Corporation, Redwood Shores, Calif. Sets of two or more computers are grouped into real application clusters. The clusters harness the processing power of multiple interconnected computers to provide a single robust computing environment. Within each cluster, all nodes concurrently execute transactions against the same database to synergistically extend the processing power beyond the limits of an individual component. Upon the mounting of the shared database, the real application cluster processes a stream of concurrent transactions using multiple processors on different nodes. For scale-up, each processor processes many transactions. For speed up, one transaction can be executed spanning multiple nodes.
Cluster databases provide several advantages over databases that use only single nodes. For example, cluster databases take advantage of information sharing by many nodes to enhance performance and database availability. In addition, applications can be sped up by executing across multiple nodes and can be scaled-up by adding more transactions to additional nodes. Multiple nodes also make cluster databases highly available through a redundancy of nodes executing separate database instances. Thus, if a node or database instance fails, the database instance is automatically recovered by the other instances which combine to serve the cluster database.
Cluster databases can be made more highly available through integration with high availability frameworks for each cluster. The inclusion of these components provides guaranteed service levels and ensures resilient database performance and dependable application recovery. Organizationally, individual database servers are formed into interconnected clusters of independent nodes. Each node communicates with other nodes using the interconnection. Upon an unplanned failure of an active database server node, using clusterware, an application will fail over to another node and resume operations, without transaction loss, within a guaranteed time period. Likewise, upon a planned shutdown, an application will be gracefully switched over to another node in an orderly fashion.
The guarantee of service level thresholds is particularly crucial for commercial transaction-based database applications, such as used in the transportation, finance, and electronic commerce industries. System downtime translates to lost revenue and loss of market share. Any time spent recovering from a system failure is measurable in terms of lost transactions. Consequently, high availability systems budget a set time period to help minimize lost revenue due to unplanned outages. High availability systems also budget for planned service interruptions.
Effective failover detection requires fast notification of unexpected database instance termination. In the prior art, two types of monitors are used to detect database instance termination. A looks-alive monitor is a lightweight process that issues an alert when the database instance terminates. An is-alive monitor is a heavyweight process that issues an alert when the database instance is not functioning or performing properly. The termination notification functions as a signal to the surviving nodes to immediately transfer the application services provided by the failed node to a pre-selected standby node. Consequently, these monitors must be fast enough to detect the termination of a database instance as soon after a termination event occurs as is possible.
Due to the less expensive mechanisms used, looks-alive monitoring is executed more frequently than is-alive monitoring. Typically, looks-alive monitoring is performed in two ways. First, the process table can be read on every invocation to tally those mandatory processes that pertain to the monitored database instance. Periodically, the process table is scanned to detect any terminations. The approach is inherently late. The cycle for waking up and checking the process table typically occurs every 30 seconds. Thus, notification can also be delayed for up to 30 seconds and can exceed the high availability time budget. Moreover, this approach is poll-based and continuously consumes processing resources. Finally, the scan is subject to hanging, which is most likely to occur when the system failure is in a failure state and when the looks-alive monitor is most needed to initiate a failover.
Second, process identifiers can be monitored instead of the process table. The process identifiers are obtained from the operating system when the database instance starts and only these process identifiers are checked on every invocation of the detection process. Although simpler, this approach is crude and ad hoc. For example, in a UNIX operating system environment, the list of process identifiers is established and a “kill -0” command is periodically executed against each process identifier. In response, an error message is generated by each process. However, since error messages can also be raised in other situations, this approach is only marginally accurate. Moreover, this approach is also poll-based and subject to hanging and therefore suffers similar drawbacks as the process table approach. Similarly, notification can be delayed 30 seconds or longer and can also exceed the high availability time budget.
Therefore, there is a need for an approach to detecting the termination of a database instance within the constraints of a high availability time budget. Such an approach must provide substantially immediate notification without continuously consuming processing resources.
There is a further need for a lightweight process for immediately detecting the termination of a database instance. Such an approach would take advantage of any internal monitors inherent in a parallel clustering framework.